Skip to content

fix(migration): stream exports and externalize historical large files#1238

Open
wafo210715 wants to merge 2 commits into
proma-ai:mainfrom
wafo210715:fix/migration-export-oom
Open

fix(migration): stream exports and externalize historical large files#1238
wafo210715 wants to merge 2 commits into
proma-ai:mainfrom
wafo210715:fix/migration-export-oom

Conversation

@wafo210715

Copy link
Copy Markdown
Contributor

Motivation

Backup export had repeatedly failed with a generic Electron IPC-style “Error invoking remote method 'migration:exportV2': RangeError: Array buffer allocation failed” message.

(以下截图为商业版)
fd1bcc6e27f6f316e60d10d9555be636

After multiple controlled tests, the failure was narrowed down to historical large files already present in several Agent session workspaces before 64a73f301 (fix large file attachment rendering, 2026-05-13). These legacy files included video recordings, geospatial TIFF data, and other large assets that had been physically copied into ~/.proma/agent-workspaces.

The existing large-file attachment flow prevents newly added files over 100MB from being copied into session workspaces. It does not address these pre-existing copies, which could still make backup export slow, fail, or exhaust memory.

Summary

This PR hardens migration exports against large workspace data and provides a safe migration path for historical files that were copied into Agent session workspaces before 64a73f301 (fix large file attachment rendering, 2026-05-13) introduced the current path-backed large-file attachment flow.

Export reliability

  • Replace in-memory AdmZip export creation with streaming archiver output to avoid OOM on large workspaces.
  • Skip known generated/system directories during workspace-data export, including node_modules, .git, build outputs, caches, and virtual environments.
  • Keep session workspace data opt-in instead of exporting it by default.
  • Apply the same 100MB threshold used by the existing attachment flow as a final export guard.

Historical large-file migration

When users opt into exporting session workspace files, Proma now:

  • Scans indexed Agent session directories and workspace-files for files larger than 100MB.
  • Shows a confirmation with the number of files, reclaimable disk space, and the external attachment destination.
  • Safely migrates confirmed files to ~/Documents/Proma-attachments by default, preserving workspace/session directory structure.
  • Uses copy + SHA-256 verification + atomic publish + attachment registration + source deletion.
  • Rolls back the attachment registration and keeps the original source file if the migration cannot complete safely.
  • Registers migrated files as existing workspace- or session-level attachedFiles, so they remain available to the Agent without residing under ~/.proma.
  • Preserves archived-session state while registering migrated files.

Backup import compatibility

  • Include session-level attachedDirectories and attachedFiles in import path checks, not only workspace-level config.
  • Apply path mappings to imported Agent session metadata for both v1 and v2 backup formats.
  • Rename the import UI section from “附加目录处理” to “附加路径处理”.

Manual validation

我手动重启了,然后测试了备份。

(以下截图为开源版测试 based on 0.15.3)
e75a6f56757d29933d3f0128505234b2
89de7ae8516bdfd897e8b063c6395db7
f98d84fec173ebf0ae28dd033ae2fece

然后我又添加了三个视频,他们都成为了附加文件,值得注意的是,历史的大文件,去了备份的时候指定的目录,新增的文件延续 Erlich 的设计。

0d0545d7edd495c57f7bbe7d652e8695

这两个差异目前觉得不是很优雅,但我个人觉得从 UI 来看,没问题。且我再一次选择备份的时候,大文件都不会影响备份。

26b47fe9372e50d30ba4b2242b104d57

Manual validation migrated two historical video files and reclaimed 23GB from ~/.proma. After restarting Proma, three newly added large videos continued to use the existing path-backed attachment flow and remained visible in the session attachment panel.

Known limitations / follow-ups

  1. External large attachments are not packaged or compressed into .proma-backup. This PR treats them as user-managed files under ~/Documents/Proma-attachments. Selective sidecar packaging/compression can be considered separately.

  2. Historical and newly added large files currently reach the same attachment semantics through different physical paths.

    • Historical files are migrated out of old session/workspace directories into the new managed attachment root during export.
    • Newly added large files already use the path-backed attachment behavior introduced in 64a73f301.

    The UI behavior is coherent and subsequent backups are unaffected, but the two storage origins are not yet fully unified. A future cleanup could make this physical layout more elegant without changing the attachment contract.

Validation

  • Added focused coverage for:
    • workspace/session large-file scanning
    • blocked-directory and symlink exclusion
    • safe externalization and attachment registration
    • archived session preservation
    • session-level attachment path mapping during backup import
  • bun run typecheck
  • Focused migration tests
  • bun run electron:build

- AdmZip → archiver 流式写入,避免大工作区导出时全量载入内存
- 新增 MIGRATION_BLOCKED_DIRS,跳过 node_modules/.git/dist 等构建目录
- 新增 MAX_EXPORT_FILE_SIZE (50MB),导出时跳过超大文件并告警
- includeWorkspaceData 默认关闭,工作区文件仅在用户勾选时导出
- MigrationSettings 增加「包含会话工作区文件」开关
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant